Word Acquisition in Neural Language Models

Abstract We investigate how neural language models acquire individual words during training, extracting learning curves and ages of acquisition for over 600 on the MacArthur-Bates Communicative Development Inventory (Fenson et al., 2007). Drawing studies word in children, we evaluate multiple predictors words’ LSTMs, BERT, GPT-2. find that effects concreteness, length, lexical class are pointedly different children models, reinforcing importance interaction sensorimotor experience child acquisition. Language rely far more frequency than but, like they exhibit slower longer utterances. Interestingly, follow consistent patterns training both unidirectional bidirectional LSTM Transformer architectures. Models predict based unigram token frequencies early before transitioning loosely to bigram probabilities, eventually converging nuanced predictions. These results shed light role distributional mechanisms while also providing insights human-like models.